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Executive Summary 


This report is the first in a series of Consumer Federation of America reports evaluating 
online services that rate local service providers. Each year, tens of millions of Americans use 
these rating services to assist their shopping for local businesses ranging from plumbers and 
electricians to auto repair shops and movers. Despite their popularity, these rating services have 
been criticized for including biased, incomplete, and inaccurate information. 


This initial report assesses Angie’s List, which is visited by more than 20 million people 
each year who access information from some 10 million individual ratings and consumer 
comments. In 2016, Angie’s List was combined with another online service, HomeAdvisor, into 
ANGIE Homeservices. However, Angie’s List continued to maintain a separate website that 
gives members access to consumer reviews of tens of thousands of local businesses. Also in 
2016, Angie’s List stopped charging members an annual subscription fee, requiring that 
members only seek information about a specific type of service and provide personal 
information, including phone number, street address, and email address. Today, Angie’s List is 
funded almost entirely by advertising, referral fees, and similar payments from the companies it 
evaluates. 


Our evaluation uncovered important limitations and biases in Angie’s List. It found that 
the service: 

e Focuses largely and increasingly on house and yard services. 

e Is based on a relatively large but diminishing number of customer reviews. 


! CFA is a nonprofit association of more than 250 consumer groups that include Consumer Reports and Consumers’ 
Checkbook. Both groups serve on the CFA Board, and CFA serves on the Checkbook Board. This report was 
initiated, researched, and written by CFA. Its findings and recommendations do not necessarily reflect those of 
Consumer Reports, Checkbook, or any other CFA member group. 
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e Gives a large majority of all service providers an A rating. 


e Rates a significant number of service providers who have very few, sometimes only 
one customer review. 

e Sometimes assigns overall grades that are not consistent with customer grades. 

e Shows some evidence of containing fake customer reviews. 

e Gives business advertisers access to personal information from customers, including 
phone numbers and email addresses, that advertisers use in their marketing. 

e Prioritizes the promotion of these advertisers by selectively recommending them to 
individual consumers as “top-rated, certified pros for your projects,” even though 
there is good evidence that these businesses usually are not the best service providers. 


Our research also concluded, however, that Angie’s List can provide value to those who 
utilize the website appropriately. Consumers would do well to keep in mind that Angie’s List’s 
recommended businesses are no better or worse than non-advertisers. Instead, as part of a more 
complete consumer search also using other sources, shoppers would benefit by reading the 
detailed consumer comments for those A-rated businesses with at least 25 of these comments, 
paying particular attention to those comments that are negative. 


Introduction 


As they seek to optimize expenditures on goods and services, consumers have access to a 
great deal of information about these products from third party sources. For many decades, 
many consumers have looked to Consumer Reports as the most reliable source of information 
about national products and sellers. Consumers have done so in part because Consumer Reports 
is a nonprofit organization funded by consumer members, not by the companies they are 
evaluating, and because the organization has a relatively large budget, over $200 million 
annually, to undertake its product evaluation. 


At the local level there is only one significant nonprofit organization, funded by 
consumers with a full-time staff, that evaluates local services ranging from car repair shops to 
plumbers to kennels. That organization, Consumers’ Checkbook, was founded in 1974 and now 
publishes Checkbook Magazine in seven major urban areas — Washington D.C., Bay Area, Puget 
Sound, Chicago, Minneapolis, Philadelphia, and Boston. But only about 100,000 consumers 
subscribe to this service. A far larger number of shoppers — tens of millions annually -- consult 
the online reviews of individual consumers that are collected and presented by numerous for- 
profit organizations. The most widely used are ratings published by large companies such as 
Google and Yelp. 


E<. OU OUE] 
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Despite their popularity, the ratings published by these for-profit organizations have 
received sharp criticism. Researchers have documented that a significant percentage of the 
reviews on some websites are “fake.”” These fake comments and ratings have been submitted 


either to promote or undermine the business under review. Critics have also charged that 
funding of these websites by advertisers can bias the evaluations and how they are presented. 
Websites may screen out critical reviews, provide only advertisers the chance to respond to these 
reviews, give the listings of advertisers favorable treatment, or supply advertisers personal 
information about users that allows marketing to them. Critics have also noted that each review 
largely or wholly represents just the experience of the consumer with that service provider, not a 
comparison of that provider with other companies offering the same service. If the consumer has 
an unduly critical or uncritical approach to their evaluation, their review may be misleading.’ 


Given these criticisms and the increasing reliance of consumers on these online reviews, 
CFA is beginning an evaluation of the most important websites. We have selected Angie’s List 
for our first assessment because of its longstanding popularity and reputation. Some view 
Angie’s List as the local consumer services counterpart of Consumer Reports. As one financial 
journalist wrote: “For more than 20 years, consumers have been consulting Angie’s List for 
unbiased reviews of local services.”* Unlike business reviews by Yelp and Google, those of 
Angie’s List are submitted only by members and so are considered by many to be more credible 
and fairer. When a consumer googles a service category -- such as plumbers or car repair shops - 
- for a specific city, Angie’s List is likely to show up as one of the first Google listings. 


Since the late 1990s Angie’s List has built and maintained an online directory that allows 
consumer members to read and contribute to reviews of businesses providing local services. The 
website includes reviews on numerous service providers throughout the country, is widely 
known and used by consumers, has a relatively large budget, and is highly regarded by some 
stock analysts. 

e The national press assumes that readers recognize Angie’s List. For example, the 
company was even mentioned in a recent humor column of The New Yorker 
Magazine.” Angie’s List has reported that its commissioned research shows 97 
percent aided brand awareness and 61 percent unaided brand awareness.° 





? Michael Luca and Georgios Zervas, “Fake It Till You Make It: Reputation, Competition, and Yelp Review 
Fraud,” Management Science, v. 62 (January 28, 2016), 3412-3427. Elizabeth Dwoskin and Craig Timberg, “How 
Merchants Use Facebook to Flood Amazon With Fake Reviews,” Washington Post (April 23, 2018). 

3 For a sample of critical articles, see: Caroline Beaton, “Why You Can’t Really Trust Negative Online Reviews,” 
New York Times (June 13, 2018). Ethan Wolff-Mann, “Here’s Everything Wrong With Online Reviews,” Money 
(June 22, 2016). Geoffrey A. Fowler and Joseph De Avila, “On the Internet, Everyone’s a Critic But They’re Not 
Very Critical,” Wall Street Journal (October 5, 2009). 

4 Brittney Mayer, “Exclusive Savings From Top-Reviewed Companies,” Credit Card News (May 8, 2017). 

> Don Steinberg, “Disturbing Digital Coincidences,” The New Yorker Magazine (March 25, 2019). 

€ BusinessWire (September 30, 2016). Aided brand awareness is with prompting (mentioning the name of the brand 
in a question) while unaided brand awareness is without prompting (for example, asking what companies provide a 
service). 
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e Angie’s List offers services throughout the country — listing 66 “major markets” and 
reporting the participation of more than 55,000 local service providers.’ 


e Itis used by millions of consumers accessing company reviews that contain an 
aggregate total of some ten million comments. The company also claims a consumer 
base of five million households and reports 23 million service requests from 
consumers in 2018.8 

e In the area of evaluations and recommendations about consumer services, it is 
relatively large. The year before merging with HomeAdvisor in 2016, its annual 
income was $344 million. In 2018, the combined company, ANGI Homeservices, 
generated $1.1 billion in annual revenue.’ 

e Itis popular with some stock analysts. In its annual 2019 media conference for 
investors, Deutsche-Bank recommended stock purchases in three internet companies. 
One was ANGI Homeservices and another was its parent company, 
IAC/InterActiveCorp.'° 


Angie’s List has always been a for-profit company, and it has faced some tough questions 
about its relationship to business advertisers. In 2013, for instance, Consumer Reports wrote a 
critical analysis of Angie’s List and other online rating services that led to an article in Forbes 
titled, “Why Consumer Reports Says You Can’t Trust Angie’s List.”!’ In 2016, the company 
paid $1.4 million to settle three lawsuits claiming that advertising payments had biased the 
ratings, reviews, and placement of local service providers.!” In 2017, an investigative report by 
NBC News Chicago found that over two-fifths of local plumbing companies advertising “Deals” 
on Angie’s List were not licensed and that a number of these companies were “Angie’s List 
certified.” 


This report describes and evaluates the services listed on the Angie’s List website (which 
remains distinct from the HomeAdvisor website). To help explain the usefulness of this service 
to consumers, the report provides a brief history of the company, describes the typical consumer 
experience with the website, and evaluates the value of the service to consumers. That 
evaluation includes a discussion of: 

e the scope of services offered in terms of types of products and trends over time, 

e the method of evaluating service providers, 





7 Craig Smith, “14 Interesting Angie’s List Facts and Statistics,” DMR Business Statistics (January 28, 2019). 

8 See Angie’s List website, “How Does Angie’s List Work.” 

° For Angie’s List, see the investor relations section of its website, particularly its annual reports, and independent 
information sources such as MarketWatch. 

10 Lee Jackson, “Deutsche Bank Internet Analyst Has 3 Top Stock Picks Now,” 24/7 Wall St (March 14, 2019). 

1 “The Truth About Online Ratings Services,” Consumer Reports (September 2013). Maggie McGrath, “Why 
Consumer Reports Says You Can’t Trust Angie’s List,” forbes.com (September 18, 2013. 

12 Paul Tassin, “Angie’s List Class Action: Payments Accepted for Better Rankings,” Top Class Action (September 
26, 2017). 

13 Phil Rogers, “Angie’s List Investigation Yields Results,” NBC 5 News (June 22, 2017). 


Angie’s List: An Evaluation of Its Usefulness for Consumers | CFA 5 





e the accuracy and adequacy of the evaluations, particularly the grade ratings, 


e the usefulness of the evaluations in terms of identifying the best and worst businesses, 

e the influence of advertising on the evaluations, related recommendations, and 
placement of both, and 

e the consumer impact of a business model that prioritizes collecting personal 
information from consumers and then using this information in marketing products to 
them. 


The report concludes with a summary of the evaluation together with recommendations 
and cautions to consumers about using Angie’s List services. 


Research for this report began during the fall of 2018. Specific information cited in the 
report was drawn from the Angie’s List website in February and March, 2019. 


Company History 


Angie’s List was founded in 1995 (under a different name) as a self-described 
“homeowners grapevine” to allow consumers in Columbus, Ohio to file comments on and rate 
the local service providers they used. In 1996, the two founders — Bill Oesterle and Angie Hicks 
—renamed the company Angie’s List, purchased a similar service in Indianapolis, and began 
offering service in Cleveland. By 2006, more than 300,000 subscribers in 33 cities were paying 
annual fees averaging about $45 to gain access to consumer comments and grades of local 
services to which they were then urged to contribute their own comments and grades (A to F). At 
this time, while consumer subscriptions represented most of the company’s income, advertising 
revenue was generated from some listed businesses with an A or B grade. '* 


Over the next decade, aided by $25 million from investors in 2010, Angie’s List 
continued to grow. By 2015, it reported 3.2 million paid members.!> Advertising revenues grew 
even more rapidly, rising from 55 percent of revenues in 2009 to 80 percent of revenues in 
2015.'° Yet, the subscriptions and advertising were not sufficient to maintain the then-current 
level of services. Company managers reported in early 2016 that “absent additional financing or 


a potential sale transaction, Angie’s List would be required to make substantial reductions in cost 


structure.” 17 


14 The early history of Angie’s List is reported on by: John Schwarb, “Angie Hicks: The Woman Behind Angie’s 
List,” Indianapolis Monthly (Mary 2006). 

'S Verne Kopytoff, “Angie’s List Names former Best buy Executive as CEO,” fortune.com (September 8, 2015). 

16 Annual financial reports from Angie’s List, loc. cit. 

17 Jared Council, “Path to Merger for Angie’s List Full of Twists and Turns,” Indianapolis Business Journal (July 15, 
2017). 
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With the goal of expanding advertising revenue, company leaders restructured the 
company. Most notably for consumers, they dropped the pay wall that had restricted website 
access to paid subscribers. Going forward, any consumer who provided personal information, 
including a phone number and email address, could gain access to customer ratings and 
comments. The hope was that this change would generate much larger numbers of consumers 
using the service, making it more attractive to advertisers. As a result, for ordinary income the 


company became completely dependent on advertising by local service providers. These 
businesses were often charged more than a thousand dollars a year for a variety of services, 
including preferred placement on website pages for individual services and the ability to 
advertise discounts. In large markets, the price could be much higher. '® 


Company leaders also explored selling the service and, in the spring of 2016, they agreed 
to terms proposed by IAC/InterActiveCorp, a holding company for many brands. One of these 
companies was HomeAdvisor, which since 1998 had served as “a digital marketplace” 
connecting homeowners with local services providing home improvement, maintenance, and 
remodeling. IAC combined Home Advisor and Angie’s List into one company, ANGI 
Homeservices, that is traded on the NASDAQ. !? 


According to a 2019 IAC report to investors, in 2018 a top priority of the new company 
was “integrating HomeAdvisor and Angie’s List,” which it characterized as a “massive 
project.””° This integration included making Angie’s List services more like those of 
HomeAdvisor. It also included using the same questions and format for enrolling members. 
However, the HomeAdvisor and Angie’s List brands and websites currently remain separate. 
Both now focus on recommending, to an individual customer, a few service providers who then 
market their services to that customer. Angie’s List also continues to collect and maintain 
information on a wide variety of service providers and make that information accessible to all 
member customers. 


Consumer Experience 


To gain access to information about local service providers, consumers must select a type 
of service (e.g., plumbing), then provide information about their specific interest (e.g., clogged 
drain and when the service is desired) and themselves, including their name, address, phone, and 


'8 Mark Sallows, “How to Use Angie’s List — Advertising Costs & Pro Tips,” fitsmallbusiness.com (October 3, 
2017). Recently, for example, ANGI Homeservices quoted a monthly price of $800 ($9,600 annually) to a new 
small business for basic services in the Washington D.C. metropolitan market. After some pushback, it reduced the 
amount to $600 monthly ($7,200 annually). 

1 Jared Council, “Path to Merger for Angie’s List Full of Twists, Turns,” Indianapolis Business Journal (July 15, 
2017). 

20 “ANGI Homeservices Inc. Q4 2018 Earnings Conference Call Transcript,” Motley Fool Transcribers (February 8, 
2019). 
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email. Once they have provided this information, they are considered to be a member and, at any 
time, can sign in on the home page with immediate access to the information on individual 
service providers. 


The home page of Angie’s List includes a photo of Angie Hicks, a request for a type of 
service (“Find top-rated pros for your project”), and a horizontal menu of eight types of “popular 
services.” If consumers indicate a service need, as noted above, the website will ask them 
specific questions about this need and contact information including name, address, phone, and 
email. If this information is provided satisfactorily, the website will recommend several 
companies that advertise with Angie’s List. These companies are rotated among all advertisers 
so that each time a request is made, a different list may appear. The website will also provide 
access to information about individual service companies, including grades and comments from 
individual consumers. 


Scrolling down the home page brings you first to a few laudatory comments from 
customers then, in the footer at the bottom, a navigation menu. This menu includes: 

e “Find Local Businesses,” which not only asks about specific service need, but also 
lists major service categories (c. 120), states serviced (all 50 and DC), and major 
markets (c. 66). 

e “Services Near Me,” which lists some 700 service categories. 

e “How It Works,” which briefly explains the benefits of membership, how members 
can use the service, and frequently asked questions. 

e “Videos” and “Solution Center” provide general information about maintaining, 
repairing, and remodeling one’s home. 


Those gaining access to pages with detailed information about specific types of service 
will find a list of active companies with customer reviews going as far back as the mid-aughts. 
The first listings are those companies that advertise with Angie’s List. Two of these companies 
that advertise are profiled above the listings. The companies that advertise and receive prominent 
listings must maintain either an A or B grade Angie’s List says that all advertisers are “certified,” 
which includes passing a criminal background check and also an attestation regarding any 
needed licensing. These advertisers can offer “a coupon” (essentially a discounted price) for a 
specific service. After the listing of advertisers, non-advertising companies are listed by grade 
from A to F. On each page listing all providers of a specific service, two advertisers are featured 
with their discounted price for a specific service. 


Each company listing includes information about hours, how to contact, the services 
offered, overall customer ratings, and general information such as free estimates, payment 
options, senior discounts, and emergency service. There is an overall Angie’s List rating (A to 
F) and also a general consumer grade rating that is based on ratings for price, quality, 
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responsiveness, punctuality, and professionalism. In addition, consumers are given the 
opportunity to comment on the service they received. 


Angie’s List indicates that they will provide mediating assistance in the event a member 
is dissatisfied with service received. In addition, their website gives members access to pricing 
information for their particular need or interest, including a bell-shaped curve showing the price 
range and average cost, and a list of several “top rated businesses” that can do the work. All 
companies on this rotating list are advertisers. Members also have access to Deals in their Zip 
Code, discounts on specific services offered by companies listed. In addition, members can 
appeal to Angie’s List for help mediating a dispute with an advertising service provider. 


Before the 2017 merger, Angie’s List featured Silver and Gold memberships, which 
offered, for an additional fee, e-mail and phone support from “Member Care specialists.” 
However, there is now little mention and no evident promotion of these enhanced memberships. 


CFA Evaluation 


How useful can Angie’s List be to consumers? Are the ratings of service providers 
significantly biased in favor of those that advertise? Regardless of this advertising, do the 
overall Angie’s List grades accurately represent the ratings and comments from individual 
consumers? What cautions should consumers heed when using Angie’s List? How should 
consumers use Angie’s List to evaluate local services? And what other sources of information 
should they consult before deciding on a vendor? 


Our research has drawn a number of conclusions that help answer these questions. We 
have focused mainly on Angie’s List services available in the Greater Washington D.C. area. 
But we also spot-checked our findings from this market using other major markets.?! One 
advantage of our using the D.C. area is that it is served by Consumers’ Checkbook, a non-profit 





21 For example, in Dallas: 

e Consumers are using the same website in the same format as in DC. 

e The home page (and other pages in the website) drive customers to recommended services which advertise 
yet sometimes have few reviews and an A rating. The five plumbers recommended include one with 2 
reviews and one with a B rating. 

e Fora specific type of service, advertisers are always listed first, and two are always featured on every page 
for that type of service. 

e Most companies reviewed provide home services — e.g., 506 roof repairers and 378 plumbers vs. 77 auto 
body shops and 23 dermatologists. 

e A number of home services providers have hundreds of individual reviews while it is rare to find a non- 
home service provider with more than 50 reviews. 

e A large majority of companies receive an A rating — e.g., 73 of 96 plumbers listed for unclogging a drain. 
© Customer reviews peaked in 4 to 5 years ago. The reviews for one well-established plumber, for instance, 
increased from one in 2009 to 72 in 2015, then declined to 12 in 2018 and to only one in 2019. Of 94 

reviews for three popular dentists, only one was posted after 2016. 


<. E E] 
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consumer service evaluating local service providers (and described earlier). Checkbook is 
supported by its consumer members and receives no advertising, referral fees, or similar income 


from any businesses that it evaluates. Comparing the company ratings of Angie’s List with those 
122 


of Checkbook has helped in our assessmen 


Our research on Angie’s List has reached a number of conclusions that are summarized 
and documented on the following pages. 


e Avery large majority of Angie’s List companies and reviews are related to house 
and yard services. The information is much more limited on service companies 
that are not providing home services. 


There are several lists of featured service categories. The eight general categories 
prominently displayed on the website’s home page all relate to home or yard services. That is 
also true for a separate list of 30 service categories and for another list of about 140 categories 
associated with “find local businesses.” There is also on the website a very large list of some 
700 service categories with only a small number, a couple dozen, that are not associated with 
house and yard. Six of these outliers relate to pets and five to motor vehicles. By comparison, 
Checkbook has well-populated general categories not only for home services, but also for 
automotive services, health care providers, personal services, leisure and travel, stores, and 
personal services. 


Members of Angie’s List do have access to a page with a navigation menu that lists 
“health” and “more,” along with “house” and “yard,” as major service categories. However, 
with one exception, the “health” and “more” categories appear to have received little attention 
since the shift in emphasis to home services. There are very few consumer reviews after 2016 
for many of the service providers listed in the “health” and “more” categories. For example, 
among the most frequently reviewed providers: Only two of 124 reviews of Fairfax Family 
Practice, none of the 93 reviews for Herndon Family Practice, and only two of 64 reviews for the 
Washington Center for Dentistry were submitted in 2017, 2018, or 2019. The exception is those 
service providers who advertise. None of the three medical service providers just mentioned — 
nor any of the 54 dermatologists and 61 internists listed -- are advertisers. Yet, those outside 
home services that do advertise tend to list a higher portion of customer reviews after 2016. In 
the in-state moving subcategory, for instance, 87 of the 88 reviews for VKH Movers, who are 
advertisers, are dated after June 1, 2018. 


It seems unlikely that, despite its current emphasis on home services, Angie’s List would 
try to rid itself of non-home service advertisers paying annual fees that often exceed $1,000. 
Yet, given the company’s focus on home services revealed by corporate statements and by its 





22 Consumer Reports evaluated the two services in a review of online rating services published in September 2013. 
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website, it does seem unlikely that the company would prioritize marketing to non-advertising 
businesses that do not offer home services. In the future, it would not be surprising to see 
subcategories such as dermatology and internal medicine removed completely from the website. 


e The number of consumer reviews posted in the past four or five years has fallen 
sharply. If this number continues to decline, the reviews will supply less useful 
information about the value provided by individual service companies. 


In researching the Angie’s List website, it was obvious that the number of consumer 
reviews had declined in recent years. As Table 1 shows, this decline can be seen, for example, in 
annual numbers of posted reviews of a sample of home or yard services: Master Plumbing and 
Mechanical (BPM) for unclogging a bathtub drain (766 reviews), Benchmark Painting and 
Carpentry (BPC) for painting the walls and ceilings of living rooms (284 reviews), Maggio’s 
Roofing Co. (MRC) for repairing an asphalt shingle roof (149 reviews), S & S Electric (SSE) for 
installing an electric switch (106 reviews), and Moyer’s Lawn Service and Landscaping for 
maintaining a lawn (166 reviews). All these companies have a relatively large number of reports 
posted over the past nine years or longer. All currently advertise, so are important to Angie’s 
List. 


Table 1: Number of Consumer Reviews Per Year in Washington D.C. Area 


Year BPM BPC MRC 
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This table suggests the following: 


e For each company, the number of reviews rose until the early or mid-teens then 
declined. 

e For each company, the decline from their peak year to 2018 was substantial — by 90 
percent for BPM, by 72 percent for BPC, by 86 percent for MRC, by 79 percent for 
SSE, and by 82 percent for MLS. 


There are a number of factors that may explain changes in the number of company 
reviews over time, including: 
e allowing consumers free access to the website, which diminishes the feeling of 
members that they belong to an exclusive club whose members share information, 
e increased consumer fatigue from being asked to respond to a barrage of survey 
requests from various sources, and 
e weaker company efforts to secure consumer reviews. 


Regardless, a continuing decline in the number of reviews would make the overall ratings 
and the website less useful to consumers. 


e Angie’s List gives an A rating to a large majority of service providers. While 
this rating scale helps consumers avoid poor service providers, it makes it 
difficult for these customers to identify the very best businesses. 


Angie’s List gives service providers an overall grade that ranges from A to F. Table 2 
below shows the number of A grades and their percentage of all grades. 


Table 2: Number and Percentage of “A” Grades Assigned Businesses in Select Service 
Categories in Washington, D.C. Area 


Business Number of 
Category Companies 


Roof Repair 
Electrical Repair 








House Cleaning 
Painting Interior 
Plumbing Service 
Call 

Auto Body Repair 
Auto Service 
Internal Medicine 





























Dermatology 
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The table shows that a large majority of businesses receive an A grade. While such a 
grade-scale may help consumer members avoid the worst businesses, the grades themselves do 
not help these customers identify the very best service providers among the large number with a 
A rating. By comparison non-profit groups that rate businesses, such as Consumer Reports and 
Consumers’ Checkbook, give top ratings only to a minority of the businesses they evaluate. 
Checkbook gives checks for excellent service and for low price. In the Washington, D.C. area, 
only a minority of service providers receive double checks — for example, 10 percent of 


plumbers, 18 percent of electricians, five percent of house cleaners, and eight percent of auto 


body shops. When Checkbook rates only quality, the percentage increases to the neighborhood 


of 50 percent. For example, 48 percent of roofers and 57 percent of tree services have checks.” 


3 


Why does not Angie’s List make more of an effort to identify the best service providers? 


One reason may be that only those businesses with an A or B rating are allowed to advertise. 
Assigning more A or B ratings gives Angie’s List the opportunity to sell advertising to a larger 
number of service providers. 


e A number of businesses with fewer than five consumer reviews, some with only 


one review, receive an “A” rating. Some of these service providers not only 
receive a high grade but also are recommended by Angie’s List. Such limited 
information from consumers cannot justify a high grade rating, indeed any 
grade rating, let alone a recommendation. 


As explained above in the “consumer experience” section, the website home page for 
Angie’s List encourages customers to “find top-rated, certified pros for your project” by 
identifying a specific service interest or need, providing personal information, then receiving 
several recommended service providers. When we sought recommendations for five specific 


types of home services, six of the 20 businesses recommended (4 for each of the 5 services) had 


fewer than five reviews. In fact, two of the 20 had no reviews at all, while two others had only 


one review, as indicated below. Five of the six recommended businesses with a few reviews also 


had an A rating from Angie’s List. 
e Install electric outlet: one business with one review 
e Mow/maintain lawn: all businesses with at least 41 reviews 
e Unclog drain: two businesses with no reviews 
e Repair asphalt shingles: one business with one review, one with three reviews, and 
one with four reviews 
e Interior painting: all businesses with at least 21 reviews 





23 Information from Washington Checkbook website to which members have access. 
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Consumers’ responding to Angie’s List’s request for information about a specific service 
qualifies them as a member and gives them access to information about all service providers. A 
spot check of first pages that list providers of a specific type of service also revealed some pages 
with only a few consumer ratings and A grades.” 


e Install an electric outlet: four businesses with one review, two with two reviews, and 
two with four reviews 

e Mow/maintain lawn: two businesses with one review, two with two reviews, and two 
with four reviews 

e Unclog drain: three businesses with two reviews 


e Interior painting: three businesses with one review and one with two reviews 


In its general list of companies offering a service, Angie’s List lists the number of 
reviews only for advertisers. To learn the number of reviews for non-advertisers, consumers 
must access the pages of individual service providers. That makes it inconvenient for consumers 
to identify non-advertising A-listed businesses with many customer reviews. 


Similarly, we spot-checked those receiving an F grade in several service categories and 

learned that a significant percentage had few customer reviews. 

e Twelve businesses listed under interior painting received F grades. Five of these 12 
had only one customer review, one had two reviews, two had three reviews, and two 
had four reviews. 

e Nine businesses listed under toilet repair received F grades. Three of these nine had 
only one customer review, two had two reviews, and two had three reviews. 

e Twelve businesses listed under asphalt shingle roof repair had F grades. Three of 
these 12 had one review, and one other business had two reviews. 


It is unfair and misleading to give a grade to a business with only one or two consumer 
reviews. The sample size is much too small to make any reliable judgment about the service 
provider. When there are few reviews, the ratings are particularly unfair both to those providers 
with low grades and misleading to those customers who take high grades seriously. Consumers’ 
Checkbook will not consider giving a business a check for quality or price unless at least ten 
consumers have responded to Checkbook’s surveys about the business. Angie’s List does not 
necessarily have to exclude businesses with only a few ratings. It could put them in a separate 
category listed as having too few reviews to provide an overall grade. Grades and comments 
submitted by individual consumers could still be listed. 





24 For some services, all companies listed on the first page were advertisers, but for other services the page included 
both advertisers (listed first) and non-advertisers. 
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Why is Angie’s List willing to give grades to businesses with only one or two customer 
reviews? One obvious benefit for the company could be that this practice expands its pool of 
potential advertisers. Angie’s List would have more credibility if it gave grades only to 


businesses with a much larger number of customer reviews. It should also disclose, on the list of 
service providers for a type of service, the number of reviews posted for all businesses, not just 
those for advertisers. 


e Overall grade ratings for individual businesses appear to be based largely on the 
ratings and comments of customers of those businesses. Yet, the relationship 
between Angie’s List grades and the customer grades is sometimes not 
immediately apparent. Some Angie’s List grades appear to be higher than 
deserved, misleading consumers, while other grades appear to be lower than 
deserved, disadvantaging those businesses. 


The main basis of Angie’s List ratings of individual companies appears to be the reviews 
provided by customers of those companies. These consumers have the opportunity to offer 
comments and to provide an overall grade that reflects grades for price, quality, responsiveness, 
punctuality, and professionalism. However, in our research we found that the Angie’s List 
grades and the customer grades were sometimes inconsistent. Here are some examples for 
several businesses offering different types of services. Comparisons were facilitated by 
calculating, for a specific business, an average customer score after giving A ratings a 5, B 
ratings a 4, C ratings a 3, D ratings 2, and F ratings a 1. Using these average scores, for example, 
those with a score 4.5 or higher would merit an A rating while those with scores between 2.5 and 
3.4 would merit a C rating. 


Lawn Maintenance 
e F grade for business with 4 As and 1 F. At 4.2, would be a B. 
e D grade for business with 8 As, 4 Bs, 1 C, 2 Ds, and 3 Fs. At 3.7, would be a B. 
e A grade for business with 20 As, 1 B, 1 C, 2 Ds, and 5 F. At 4.2, would be a B. 


Asphalt Shingle Roof Repair 
e F grade for business with 7 As, 1 B, and 1 F. At 4.4, would be a B. 
e F grade for business with 4 As, 1 B, and 2 Fs. At 3.7, would be a B. 
e F grade for business with 9 As and 1 F. At 4.6, would be an A. 
e B grade for business with 4 As and no other grades. At 5.0, would be an A. 


Interior Home Painting 
e F grade for business with 7 As, 1 B, and 1 F. At 4.4, would be a B. 
e F grade for business with 3 As and 1 F. At 4.0, would be a B. 
e B grade for business with 26 As and 4 Bs. At 4.9, would be an A. 
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e A grade for business with 375 As, 74 Bs, 39 Cs, 27 Ds, and 32 Fs. At 4.3, would be a B. 


Electrical Work Upgrading Wiring 
e F grade for business with 7 As and 1 F. At 4.5, would be an A. 
e F grade for business with 12 As, 3 Bs, 2 Ds, and 2 Fs. At 4.1, would be a B. 
e F grade for business with 3 As and 1 F. At 4.0, would be a B. 


e A grade for business with 130 As, 16 Bs, 14 Cs, 7 Ds, and 10 Fs. At 4.4, would be a B. 


There are several possible explanations for apparent inconsistencies between customer 
and Angie’s List grades, including carelessness, inattention to the grades of non-advertisers, a 
desire to show some low grades to give all ratings more credibility, and giving advertisers the 
benefit of the doubt. In some cases, Angie’s list grades may be higher than those of customer 
reviewers because the company did not penalize service providers as severely for failing to 
return phone calls or keep appointments as for performing shoddy work. But this reason does 
not explain the large gap between businesses with an A or B grade by consumers and a D or F 
grade by Angie’s List. 


e There is circumstantial evidence that some service providers have engineered the 
submission of fake reviews. Such fraudulent behavior would be very unfair both 


to customers who are misled and to listed businesses that play by the rules. 


We cannot confirm fake reviews, especially because we do not have the identities of 
those submitting these reviews.” However, in our research we found what we believe to be 
circumstantial evidence of fake reviews that were part of business listings. The following two 


businesses currently are advertisers with an Angie’s List A rating who are Angie’s List Certified. 


The electrician has also received the Super Service Award. 


A roofer claimed to have been in business for 30 years, but its first “customer review” 
was posted on March 16, 2016, shortly after membership fees were waived. In the next five 


months, 361 reviews were posted in clusters on certain days. All those submitting reviews gave 


the roofer an A rating and all their comments were positive. The last cluster was posted on 


August 28, 2016. Since this date, 28 reviews have been posted. Customers submitting eleven of 


these reviews gave the roofer an F grade while two others assigned the company a D grade. 
Based on this evidence, it appears that the roofer engineered the submission of an enormous 
number of fake reviews in a short time period -- far more than we found for any other similar 
Angie’s List business -- but then after that, was subjected to harsh but real views from actual 
customers. 





25 “How to Spot Fake Online Product Reviews,” realsimple.com. 
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An electrician received 20 reviews from 2008 through 2014, 257 reviews from 2015 
through 2017, then only 13 reviews since then. As noteworthy about the reviews as their 
distribution over time is their lack of content. An unusual number of reviews — for example, 52 


of 131 in 2015, and 27 of 68 in 2016 — contain no comment. And most comments in the 
remaining reviews are very short — a word, a phrase, or a short sentence — and very enthusiastic. 
While a few of the 290 reviews are negative — four F grades and three D grades — a large 
majority are positive, with 253 receiving an A grade. One can speculate that between 2015 and 
2017, the electrician somehow engineered the submission of a large number of fake reviews 
while also continuing to receive a modest number of real reviews, some of which were negative. 


We cannot be absolutely certain that these two service provider listings, or any others, 
contain fake grades and comments. Yet, we believe that there is circumstantial evidence to 
suggest that the two listings above include fake reviews. And given the fact that we examined 
only a very small number of the some 10 million reviews published by Angie’s List website, we 
also believe that there are likely to be many other suspicious reviews. 


e Angie’s List does not provide reliable information about which businesses 
charge the lowest prices for a specific service. 


Angie’s list asks consumer members to comment on and grade the prices charged by their 
providers. This price information has some value, especially if the consumer alleges that a 
business overcharged them. But the information is not nearly as reliable as a careful survey of 
prices charged for the same service by a large number of businesses. Because few consumers on 
their own undertake such a survey, they are not always aware of those companies charging the 
lowest prices for a specific service. 


The lack of reliability of the Angie’s List price grades can be seen by comparing these 
grades to the price ratings that Consumers’ Checkbook determined by researching actual prices 
for the same service. We compared the grades and ratings for all companies rated by both 
Angie’s List and Checkbook in the Washington, D.C. area for three types of services — in-home 
carpet cleaning, kennels, and pest control. The Checkbook price score was for spraying for 
roaches. Table 3 summarizes what we found. The ratings are based on an average price score of 
100 for all businesses rated for that service, so scores below 100 indicate lower-than average 
prices while numbers above 100 indicates higher-than-average prices. “Number” represents the 
number of businesses with both grades and ratings for that service. 
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Table 3: Comparing Angie’s List Price Grades with Checkbook’s Price Scores 


Average 
Service Number Score Low 


Carpet Cleaning 
Grade A 
Grades B-F 








Kennels 
Grade A 
Grades B-F 
Pest Control 
Grade A 
Grades B-F 





























As is evident, there is little relationship between the Angie’s List price grades by 


consumers and Checkbook’s price scores based on surveys of actual prices. For carpet cleaning, 


there is an inverse correlation between Angie’s List grades and Checkbook’s price scores; for 


kennels, there is little correlation; and for only pest control is there a positive correlation. More 


importantly, the range of individual ratings for each service is substantial, typically more than 


100 percent. This comparison suggests that Angie’s List price grades are not a reliable indicator 


of those businesses charging low prices for a particular service. 


e Business advertisers have advantages over non-advertisers in listing placement 
characterization, and ability to lessen negative reviews. Consumers may well 
confuse Angie’s List’s promotion of advertisers with its evaluation of these 
companies. 


Now that customers are no longer paying annual subscription fees, Angie’s list depends 


S, 


almost entirely on revenue from business advertising. To what extent does this advertising bias 


the placement and even content of business listings and evaluations?”° 


First, only business advertisers are recommended as “top-rated pros,” but these 
companies are unlikely to be those providing the best service. Angie’s List recommends 
“top-rated pros” in various ways, including when consumers become members by expressing 
interest in a particular service, when consumer members request recommendations from the 





26 Most of the benefits offered advertisers can be found in a document we could not find on the internet but was sent 


to us by Angie’s List after we expressed interest in becoming an advertiser, “Let Us Help Your Business Grow.” 
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Support Center, and in emailed ads targeting consumer members.”” However, as documented 
earlier, these “top-rated pros” include companies with few reviews, include companies with B 
grades even though a large majority have A grades, and usually represent a small percentage of 
all A-rated companies in a service category. We could not find a disclosure, on any of the pages 
with these recommended companies, that the companies were advertisers. Thus, consumers may 
well assume that these recommendations are based entirely on merit. 


Second, advertisers always are listed first on listings of companies for a specific type 
of service.’ And, unlike those of non-advertisers, their listings always note that they are 
“Angie’s List Certified” and sometimes indicate that they have received Angie’s List’s “Super 
Service Award.” Angie’s List also selects two advertisers whose discounted “Deal” for a service 
is presented on each page of the listing for the type of service.” It would be very easy for 
unsuspecting consumers to assume that Angie’s List was endorsing these businesses and/or 
considered them to be the very best companies to provide a type of service. Regardless, as 
marketing research has shown, when confronted with a list of many pages, users are likely to 
spend the most time on page one. 


Third, Angie’s List biases their ratings to favor advertisers. The Angie’s List grades 
are based largely on the grade ratings and comments provided by customers of a specific 
business. Angie’s List helps advertisers secure these grades and comments.*° For new 
advertisers, they will contact a list of past customers, supplied by the advertiser, to try to secure 
their ratings. Lists submitted by advertisers are likely to include only those customers they think 
were satisfied. If the rating or comment is negative, Angie’s List offers to contact the customer 
and mediate a mitigation of their dissatisfaction and submission of a revised rating and comment. 
As aresult, all other factors being equal, businesses that do not advertise are more likely to have 
negative ratings in their listing.*! 


The combination of these individual biases creates a large bias in favor of advertising 
companies.*” Sophisticated consumers who know something about Angie’s List can work 
around this bias. But most consumers do not have this knowledge so are vulnerable. That is the 


27 Mark Sallows, “How to Use Angie’s List,” loc. cit. “Let Us Help Your Business Grow,” loc. cit. For example, a 
recent email sent by Angie’s List to an enrolled member about a particular service recommended four businesses, 
two of which had no customer reviews and one of which had only two reviews. 

28 “Let Us Help Your Business Grow.” Information sent by Angie’s List sale office in response to our request. 

2 Consumers should not take discounted prices too seriously. Not infrequently in the marketplace, vendors mark up 
prices then discount them to make them more attractive to shoppers. One may see evidence of this in the very high 
discounted prices offered by some Angie’s List businesses. 

30 According to Angie’s List sales representative in phone conversation with CFA researcher. 

3! David Segal, “The Haggler: A Complaint Registered Then Expunged,” New York Times (December 21, 2013). 
32 These biases are also suggested by the complaints of some companies that formerly advertised with Angie’s List, 
as can be seen on websites such as Consumer Affairs and Handyman Startup. 
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assumption of advertisers who think their expenditure will purchase them some advantage in 


competing with non-advertising companies. 


e Business advertisers have the ability to market to consumers who use Angie’s 
List. The highest priority of the Angie’s List website is to persuade consumers to 
request “top-rated pros” who can perform a specific service. Consumers who 
make a request supply information about specific service needs and about 
themselves, including phone number and email address. This information is 
passed on to advertisers, who often try to contact the consumers. These 
businesses are unlikely to be those that offer the best service. 


For decades, consumer educators and advocates have urged consumers to undertake their 
own consumer search for a product or service, then make contact with vendors on their own 
initiative. These consumer professionals also urge consumers not to respond to phone, email, or 
snail mail solicitations. Rarely do solicitors offer the best value in a product or service. 


Angie’s List does not initiate “cold calling” by service providers but it does facilitate 
their solicitations. To gain access to the information about individual companies on the Angie’s 
List website, a consumer must provide personal information, including their phone number and 
email address. Angie’s List passes on this information to business advertisers, who then are free 
to market their services to the consumer. And it may also sell this information to other 
businesses. For example, when we requested information about home movers, we very quickly 
received several calls from companies that indicated they had received our information from 
Home Advisor. This marketing is permitted by the Angie’s List Privacy Policy, found on the 
website, to which users consent “by using the site and services.” If “you do not agree to the 
terms and conditions of the privacy policy... you may not be able to use certain parts or features 
of our site and services, and in some instances, this may necessitate the revocation of your 
membership.” 


The solicitations are most likely to take place when a consumer requests information 
about a specific service, such as unclogging a drain. Angie’s List recommends several (rotating) 
companies, all advertisers, and gives them personal information about that consumer. Generating 
and profiting from these consumer service requests is currently a high priority of Angie’s List. 

In a recent interview with large investors, the company’s chief financial officer said: “We have a 
large sales force ... and it’s much more weighted toward consumers to drive service 
requests....We’ve got a lot more efficient with our spend and our ability importantly to monetize 


every one of those service requests.” 3° 





33 ANGI Homeservices Inc. Q4 2018 Earning Conference Call Transcript, loc. cit. 


E<.. OOO] 
Angie’s List: An Evaluation of Its Usefulness for Consumers | CFA 20 





Advertising companies may not be those that provide the best service. While all 
recommended companies have overall A or B grades, as explained earlier, these ratings may not 
be fully merited. The grades may be based on only a few customer ratings, or may have been 


inflated by their careful solicitation and curating, or even submission of fake reviews. 
Regardless, in most service categories, only a minority of those with A grades are advertisers. 


Supporting evidence that the companies promoted by Angie’s List are not always the best 
ones for consumers can be found by comparing these advertisers to those local service 
companies which receive Checkbook Magazine’s highest rating — a check for quality and a check 
for price. As noted earlier, Checkbook is a nonprofit organization that evaluates service 
providers in seven urban areas and accepts no funds from the companies that it evaluates. It will 
not consider check-rating a business unless it has received reviews from at least ten customers of 
that business. And its ratings are based not just on customer reviews but also on research by 
Checkbook staff. We selected four of its house and lawn services — tree services, roofers, 
plumbers, and electricians — that Angie’s List also evaluated for the same Washington, D.C. 
consumer. 


As can be seen below, very few A-rated businesses that advertised with Angie’s List 
received a check for quality and/or price by Consumers Checkbook. 


Tree Services: There were 10 A-rated advertisers among Angie’s List (for tree 
trimmers) and 63 tree services checklisted for quality by Washington Checkbook. Only one 
Angie’s List advertiser was check-listed by Checkbook. 


Roofers: There were 19 A-rated advertisers among Angie’s List roofers (who repair 
asphalt shingles) and 43 roofers checklisted for quality by Washington Checkbook. Only one 
Angie’s List advertiser was check-listed by Checkbook. 


Plumbers: There were six A-rated advertisers among Angie’s List plumbers (for a 
general service call) and 21 plumbers check-listed for both quality and price by Washington 
Checkbook. None of Angie’s List advertisers was checklisted for both quality and price by 
Checkbook. 


Electricians: There were seven A-rated advertisers among Angie’s List electricians (for 
repairing a light switch or outlet) and 14 electricians checklisted for both quality and price by 
Washington Checkbook. None of the Angie’s List advertisers was check-rated for quality and 
price by Checkbook, and only one of the advertisers was checklisted just for quality. 
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In general, Angie’s list includes a larger number of service providers for home service 
categories and a much larger number of customer reviews than does Checkbook. Rating 
differences cannot be explained away by the fact that Angie’s List includes some businesses that 


do not appear well-established (1.e., few reviews), sometimes specialize in services outside the 


reviewed service category, or are located far from Washington, D.C. Thus, it is noteworthy that 


so few of Angie’s List’s A-rated advertisers were check-rated by Checkbook — only two of 42 
service providers in the four service categories. The most obvious explanation for this absence 


is 


that Angie’s List gives an A rating to most services listed and that the few who advertise usually 


are not among the best service providers. 


Summary and Recommendations 


This report has focused most attention on the limitations, weaknesses, and biases of 
Angie’s List. It has found that the service: 

e Focuses largely and increasingly on house and yard services. 

e Is based on a relatively large but diminishing number of customer reviews. 

e Gives a large majority of all service providers an A rating. 


e Rates a significant number of service providers who have very few, sometimes only 


one customer review. 
e Sometimes assigns overall grades that are not consistent with customer grades. 
e Shows some evidence of containing fake reviews. 


e Does not provide reliable information about businesses that charge the lowest prices 


for a specific service. 

e Gives business advertisers advantages in listing placements, characterization, and 
ability to mitigate negative customer reviews. 

e Gives advertisers access to personal information from customers, including phone 
numbers and email addresses, that advertisers use in their marketing. 

e Prioritizes the promotion of these advertisers by selectively recommending them to 
individual consumers as “top-rated, certified pros for your projects,” even though 


there is good evidence that these businesses usually are not the best service providers. 


As the report explains, the content and presentation of information on the Angie’s List 
website appears to be strongly influenced by the company’s dependence on advertising by 


companies being evaluated. Since its founding in the mid-1990s, Angie’s List has moved away 
from dependence on paid consumer subscriptions to advertising by some of the listed businesses. 


The purchase of Angie’s List by the holding company IAC in 2016, which combined the 


company with HomeAdvisor to form ANGI Homeservices, raised additional questions about its 


commitment to providing helpful consumer advice as opposed to monetizing service 
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relationships. An important indicator of its future priorities will be the extent to which it 
generates a large number of customer reviews with detailed comments about service providers. 


Having said all this, Angie’s List can offer value to consumers who are willing to divulge 
personal information, including phone number and email address, then be marketed to by 
advertisers. Most consumers in the country do not have access to a well-established, nonprofit 
organization that evaluates local service providers more objectively. These consumers can 
benefit by selectively and carefully consulting Angie’s List customer reviews, especially the 
comments. 


In this “consultation,” consumers should ignore companies recommended or profiled by 
Angie’s List. Consumers should then: 

e Give greatest weight to the customer reviews of all A-rated businesses with at least 25 
recent customer reviews. 

e Pay closest attention to comments that describe in detail the service performed and 
customer satisfaction with it. These reviews are most likely to be genuine and 
helpful. 

e Pay particular attention to any negative comments. Because of the way these 
customer reviews are collected and curated, businesses offering poor service may 
well have a large majority of positive reviews. 


But consumers should not, in their product search, rely too heavily on information from 

the Angie’s List website. 

e The experience of friends, family members, and acquaintances may be helpful. 

e A Google search could reveal consumer criticism or bad business practices. Search 
not just by the company’s name but also by the name plus terms such as “consumer 
complaints,” “bad service,” “rip-off,” and/or “sucks.” 

e Interviews with a least three promising service providers will usually be helpful in 
making a final decision. And if the work is expensive, getting competitive bids from 
these providers could prove invaluable. 
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